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Technical  Accomplishments 

Under  this  grant  research  was  conducted  at  both  Adaptive  Systems  Inc. 
(ASI)  and  the  Oregon  Graduate  Institute  (OGI).  ASI  is  developing  hardware 
implementations  of  models  derived  from  studies  of  olfactory  cortex.  For 
these  implementations  the  models  were  mapped  onto  the  general-purpose 
neurocomputer  developed  at  ASI.  Research  at  ASI  is  also  directed  towards 
direct  silicon  implementation  of  biologically  faithful  cortex  models. 

Research  at  OGI  is  concerned  with  the  computational  capability  and  the¬ 
oretical  aspects  of  these  models,  as  well  as  with  modihcations  that  enhance 
functionality.  We  applied  cortex-inspired  models  to  speech  recognition  prob¬ 
lems  and  pursued  issues  of  model  convergence  and  computational  efficiency. 

During  the  last  year  of  the  grant  we  explored  two  new  lines  of  inquiry. 
The  first  deals  with  the  dynamics  of  stochastic  learning  systems,  and  the 
second  with  locally-linear  data  encoding.  These  are  described  in  more  detail 
below. 

Implementation  on  High-level  Simulator  (OGI)  The  abstract  cortex 
model  [1]  was  implemented  in  ASI’s  high-level  simulator.  This  was 
the  first  step  in  mapping  the  algorithm  to  ASI’s  neuro-computer.  This 
implementation  provides  a  template  for  the  direct  micro-code  imple¬ 
mentation  on  the  chip.  In  addition,  this  simulator  provided  us  with 
a  vehicle  for  preliminary  studies  of  the  clustering  and  classification 
ability  of  the  model. 

I  Thi'.  d.'”'nri,'al  ri' 

foi  public  roio  )  .'  ■  '  .J  ,  i(  ■  1 

j  diitiibution  is  i 


92  y  01  005 


92-24200 


DTIC 

ELECTE 
SEP  0  8 1992 

A 


Implementation  in  Micro-Code  (ASI)  We  have  completed  a  direct  micro¬ 
code  implementation  of  the  model  for  the  ASI  neuro-computer.  The 
code  includes  embellishments  to  the  algorithm  developed  by  the  re¬ 
search  group  at  OGI.  Details  of  the  implementation  appear  in  [2]. 

Computational  Capability  (OGI)  In  [3]  we  have  shown  that  the  ab¬ 
stract  model  described  in  [1]  and  implemented  on  the  ASI  neurocom¬ 
puter  is  a  neural  implementation  of  multistage  vector  quantization 
[4,  5].  This  architecture  has  a  significant  storage  advantage  relative 
to  conventional  neural  clustering  and  vector  quantization  (VQ)  algo¬ 
rithms. 

Standard  neural  VQ  algorithms  require  one  neuron  for  each  cluster 
point.  That  is,  each  neuron  responds  to  a  specific  region  of  the  input 
space.  This  defines  the  receptive  field  of  the  neuron. 

The  multi-stage  architecture  defines  cluster  points  by  combinations 
of  neural  receptive  fields.  For  example,  a  cortex  model  with  three 
hierarchical  levels  of  four  neurons  each  generates  64  distinct  cluster 
points  using  only  12  neurons.  A  conventional  competitive  learning 
algorithm  requires  64  neurons  to  define  the  same  number  of  cluster 
points.  Our  experiments  confirm  that  the  cortex  model  when  used 
for  encoding  and  classifying  phonemes  provides  lower  error  per  neuron 
than  either  conventional  or  tree-structured  algorithms  [3]. 

Naturally  one  does  not  get  something  for  nothing,  and  there  is  a  trade¬ 
off  inherent  in  the  multi-stage  structure.  Since  each  neuron  in  the 
model  participates  in  defining  several  clusters,  the  latter  cannot  be 
independently  oriented.  However,  the  disadvantage  of  this  constraint 
appears  to  be  overwhelmed  by  the  advantage  gained  by  the  combina¬ 
torial  efficiency. 

Model  Dynamics  (OGI)  We  began  studies  of  the  dynamics  of  self-organizing 
networks  under  a  previous  ONR  grant  and  completed  that  work  under 
the  present  grant.  (Cliff  Lau  served  as  science  officer  for  the  previous 
grant.  No.  N00014-88-K-0329.)  We  introduced  tools  from  bifurca¬ 
tion  theory  to  treat  the  dynamics  of  learning  in  networks  with  both 
Hebbian  and  anti-Hebbian  synapses  and  recurrent  lateral  connections. 
Our  studies  identify  the  bifurcation  types  and  thus  provide  analytic  de¬ 
scriptions  of  the  location  of  equilibria  and  limit  cycles  in  the  learning 
behavior. 


2 


Our  paper  describing  this  work  was  among  fewer  than  6  %  of  the 
submitted  papers  chosen  for  oral  presentation  at  the  1990  Neural  In¬ 
formation  Processing  Systems  conference  at  Denver,  Colorado  [6].  The 
work  was  recently  published  in  Network  :  Computation  in  Neural  Sys¬ 
tems  [7]  and  is  the  subject  of  invited  talks  at  the  1990  International 
Society  for  the  Systems  Sciences  and  the  1991  SPIE  Conference  on 
Adaptive  Signal  Processing. 

Stochastic  Learning  Dynamics  (OGI)  In  stochastic  learning,  param¬ 
eter  (synapse)  adjustments  are  made  after  each  data  presentation, 
rather  than  after  a  “batch”  of  data  has  been  presented.  The  Lynch- 
Granger  olfactory  cortex  models  are  examples  of  stochastic  learning 
systems.  Much  of  the  literature  in  stochastic  learning  is  concerned  with 
proving  convergence  to  local  optima.  There  is  also  a  body  of  literature 
concerned  with  optimizing  the  convergence  rate  to  local  minima. 

Drawing  on  tools  from  statistical  physics  and  the  theory  of  stochas¬ 
tic  processes,  we  have  developcf'  an  analytic  approach  to  study  the 
ensemble  properties  of  stochastic  learning.  The  approach  allows  us 
to  calculate  transient  and  equilibrium  probability  distributions.  We 
applied  the  formalism  to  study  the  statistical  distribution  of  times  re¬ 
quired  to  pass  between  basins  of  different  local  optima,  equilibrium 
distributions,  and  optimal  learning  rate  schedules. 

This  work  was  published  in  the  1992  IJCNN  [8]  and  presented  at  the 
1992  Snowbird  Conference  [9].  We  have  two  papers  accepted  for  the 
1992  NIPS  meeting,  one  for  oral  presentation  [10],  and  one  for  poster 
presentation  [11]. 


Locally-Linear  Encodings  (OGI)  The  second  line  of  inquiry  extends  the 
piece-wise  constant  encoding  produced  by  algorithms  like  the  Lynch- 
Granger  cortex  model.  In  order  to  achieve  more  accurate  representa¬ 
tions,  we  combine  local  linear  models  with  vector  quantization  (VQ). 
The  resulting  algorithm  produces  a  non-linear  dimension  reduction 
that  is  superior  to  both  VQ  and  the  globally  linear  encoding  produced 
by  principal  component  analysis  (PCA). 

The  only  competing  algorithm  for  non-linear  dimension  reduction  uses 
five-layer  feed-forward  networks  trained  to  perform  an  identity  trans¬ 
formation  between  tho  input  and  output  layers.  Applied  to  speech 
data,  our  algorithm  runs  about  five  times  faster  than  five-layer  net¬ 
works,  and  produces  more  accurate  encodings. 
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